10 research outputs found
SentiBench - a benchmark comparison of state-of-the-practice sentiment analysis methods
In the last few years thousands of scientific papers have investigated
sentiment analysis, several startups that measure opinions on real data have
emerged and a number of innovative products related to this theme have been
developed. There are multiple methods for measuring sentiments, including
lexical-based and supervised machine learning methods. Despite the vast
interest on the theme and wide popularity of some methods, it is unclear which
one is better for identifying the polarity (i.e., positive or negative) of a
message. Accordingly, there is a strong need to conduct a thorough
apple-to-apple comparison of sentiment analysis methods, \textit{as they are
used in practice}, across multiple datasets originated from different data
sources. Such a comparison is key for understanding the potential limitations,
advantages, and disadvantages of popular methods. This article aims at filling
this gap by presenting a benchmark comparison of twenty-four popular sentiment
analysis methods (which we call the state-of-the-practice methods). Our
evaluation is based on a benchmark of eighteen labeled datasets, covering
messages posted on social networks, movie and product reviews, as well as
opinions and comments in news articles. Our results highlight the extent to
which the prediction performance of these methods varies considerably across
datasets. Aiming at boosting the development of this research area, we open the
methods' codes and datasets used in this article, deploying them in a benchmark
system, which provides an open API for accessing and comparing sentence-level
sentiment analysis methods
Smoke detector: Cross-product intrusion detection withweak indicators
The central task of a Security Incident and Event Manager (SIEM) or Managed Security Service Provider (MSSP) is to detect security incidents on the basis of tens of thousands of event types coming from many kinds of security products. We present Smoke Detector, which processes trillions of security events with the Random Walk with Restart (RWR) algorithm, inferring high order relationships between known security incidents and imperfect secondary security events (smoke) to .nd undiscovered security incidents (fire). By finding previously undetected incidents, Smoke Detector's RWR algorithm is able to increase the MSSP's critical incident count by 19% with a 1.3% FP rate. Perhaps equally importantly, our approach offers significant benefits beyond increased incident detection: (1) It provides a robust approach for leveraging Big Data sensor nets to increase adversarial resistance of protected networks; (2) Our event-scoring techniques enable efficient discovery of primary indicators of compromise; (3) Our con.dence scores provide intuition and tuning capabilities for Smoke Detector's discovered security incidents, aiding incident display and response